Wrap Up!

Hi everyone,
This blog post marks the official end of GSOC 2016, however, I will continue my association with Mailman and keep working with same zeal. Below are the links to the branches on which I worked during the GSOC coding period:

1: Phase1 branch:
https://gitlab.com/_Harshit_/mailman/commits/phase1
This is the first branch in a series of four branches(two in mailman core, one in mailman client and one in postorius).
Goals achieved:
1: Allow for storing of styles in database:
For serving this purpose, I have introduced two new models:
1: Stylet
2: Style
Stylets are a partial set of styleable attributes that can be used to compose styles which are a stack of stylets such that stylet at the top overrides the values of the common attributes.
2: Allow for the automatic updation of the settings of the mailing list as soon as the style being used by it is updated:
I have used two of the SqlAlchemy's session level events for this purpose 'after_flush' and 'after_flush_postexec'. Both of the events are fired as soon as a flush occurs. In the 'after_flush', we collect all the stylet/style objects in which there are any net changes. Since this event doesn't allow us to emit further SQL until the flush has been performed, hence I have used after_flush_postexec which is triggered after the flush the has been executed. In this event we update all the mailing list which use the updated stylet/style.
3: Change the style used by a mailing list after its creation:
This branch allows for the changing of the style of the mailing list after its creation. The mailing list whose style is changed will start to follow the new style such that any changes made to any style(or any stylet of which this style is composed) will be reflected in the mailing list. However, one important thing to be noted is that there is only a loose coupling between a mailing list and the style it uses in the sense that if a mailing list was using a style which for example defined a value of True for the 'advertised' attribute and later on it switches to another style which defines no value for the 'advertised' attribute then the mailing list will continue to have a value of True for the 'advertised' attribute until the new style defines a value for this attribute.
4: Make a stylet/style immutable:
This branch allows for making a stylet/style immutable. A style once set immutable can't be modified. For this purpose I have overridden the 'setattr' method of the style and stylet model and if a style has been set immutable we raise a StyleUpdateError.
5: Porting the default stylets/styles:
I have ported the default stylets and styles. The stylets/styles are added to the database to the automatically during startup if the stylet table(and hence the style table) is empty.
Things yet to be done:
1: Documentation:
The stylet/style update system and the loose coupling between the styles and the mailing list is too complex and will need extensive documentation.

2: Phase2 branch:
https://gitlab.com/_Harshit_/mailman/commits/phase2
This is the second branch in the series.
Goals achieved:
1: Expose stylets via REST API:
This branch exposes stylets via REST API which means users can now view, edit and delete existing stylets as well as create new stylets.
2: Expose styles via REST API:
This branch also exposes styles via REST API which means just as stylets, users can view, edit and delete existing styles as well as create new one. Not only this, they can also inherit from an exsting style.
3: Remove the existing way of changing list settings:
Since almost all the settings of a mailing list are managed via styles and they can now be updated dynamically updated, hence, this branch removes the existing ListConfiguration class and makes manual editing of list settings non-recommended way of changing list settings.
4: Allow for changing a list's style via REST API:
Earlier, REST API allowed users to apply style to a list only at its creation time. This branch allows users to change the style of a mailing list via REST API.
Things yet to be done:
1: Documentation:
All the REST endpoints needs extensive documentation.

3: Phase3 branch:
https://gitlab.com/_Harshit_/mailmanclient/commits/phase3
This is the third branch in the series.
Goals achieved:
This branch introduces some changes to the exisitng python bindings(mailman client) in accordance to the changes introduced in the REST API in phase2 branch.
Things yet to be done:
1: Testing
2: Documentation

4: Phase4 branch:
https://gitlab.com/_Harshit_/postorius/commits/phase4
Goals achieved:
1: Provide an interface to manage stylets:
This branch introduces a new interface using which users can easily view, edit, copy or delete stylets.
2: Provide an interface to manage styles:
This branch introduces a new interface using which users can easily view, edit, copy, inherit or delete styles.
3: Remove the interface for changing the list settings.

Screenshots: https://mail.python.org/pipermail/mailman-developers/2016-August/025792.html

Things yet to be done:
1: An interface using which users with higher privileges can defer attributes to users with lower privileges.
2: Modify the existing interface so that users can apply styles to a new list or to an existing list.
3: Testing.
4: Documentation.

Read my other blog posts for more info.

Thanks,
Harshit Bansal

Post Midterm Evaluations

Hi readers,
Finally Midterm evaluations have passed and fortunately I have successfully passed them. As I mentioned in my previous post, phase1 of the project was about to complete as only the testing part was remaining and I am happy that the code is now passing all the tests successfully(sqlite is still failing). Although the code is passing all the tests successfully now but as always there is a problem. Earlier, the styles were stored as python files and all that style manager had to do was to look in certain paths to check for any new styles. As styles are now stored in the database and the testing layer resets the database after each test, we now need to populate the default styles into the database after each test(as create_list function in app/lifecycle.py which is being used in a number of tests requires the 'Legacy Default' style). This has made running tests painfully slow. Earlier running all the tests took about 15 minutes to complete and now it is taking approximately 55 minutes to get completed. Finding a solution to this problem is now at the top of my to do list and I hope that I will be able to find a solution soon.
Besides this, I have started to work on the phase2 of the project(REST API). First step in this direction is to decide where these resources will live in the tree. I am having some ideas and soon I will be discussing them with Barry and others.
There is one more that I forgot to mention that I have again made a small change in the design. As mentioned in earlier posts, styles which used to represent a complete set of attributes are now just a stack a stylets don't need to contain all the styleable attributes.
I am very much excited and ready to get most out of this summer. Stay connected for more.

Thanks,
Harshit Bansal

First Progress Report & Future Plans

Hi readers,
This post is going to be a short one focusing on giving an overview of the work done from the starting of the coding period to the work foreseen up until the next week that is the end of the mid term evaluations. It has been quite some time since the coding period has started and my project has made an ample amount of progress during this period. So let's get started:
I had divided the project in three phases:

• Phase1 : Changes in mailman core.
• Phase2 : Implementing the REST API.
• Phase3 : Front end(Postorius).

Of these I had proposed to complete Phase1 by the end of Midterm Evaluations(i.e., 27th June) and with about a week in hand I am pretty sure that I will be able to complete it according to the proposed timeline.
Work done so far:

• Add new IStyleable Interface(Done): https://gitlab.com/_Harshit_/mailman/tree/IStyleable_Interface

• Design and implement:

1.  IStylet interface(Done): https://gitlab.com/_Harshit_/mailman/tree/IStylet_Interface
2.  IStyletManager interface(Done): https://gitlab.com/_Harshit_/mailman/tree/IStyletManager_Interface
3.  IStyle interface(Implementation part done, testcases remaining): Will be done as soon as the two default styles get ported.
4.  IStyleManager interface(Implementation part done, testcases remaining):Will be done as soon as the two default styles get ported.

• Port the two default styles: Currently working on this.

• Refactor and make some changes in the implementation IMailingList: Almost done. Requires testing which will be done as soon as the above mentioned three works get finished.

Future Plans:

From the next month(coding period 2, three days reserved for tiding up code, make up for any delay and to recieve approval from mentor), I am planning to start working on the implementation of the REST API and will try to complete it within about one and a half week after which I will start working on Phase3(Postorius). I am thinking of submitting the two merge requests related to the phase1 and phase2 at the end of phase2 so that if I feel a need to make some changes in the core while implementing the REST API I would be able to do that.

Stay connected for more.

Thanks,
Harshit Bansal

New design explained

Hi readers,
This post mainly focuses on the design that my project now follows. Firstly, I am going to give you an overview of my previous design and then going to mention the changes that the new design introduces. So let's get started:

So what was the Older design?

This was the design that I proposed earlier in my proposal. A style was considered to be describing a complete set of styleable of attributes. A user can either create a new style or inherit from an existing style and make changes as per the requirements. The inheriting style will also reflect any changes made in the parent style. A style was supposed to have three levels of viewabilities:

1.  System-wide viewability: Style is visible to all the lists.
2.  Domain-wide viewability: Style is visible to the lists present in a specific domain to which the style is associated.
3. List-wide viewability: Style is visible to only a particular list to which the style is associated.

Not only this all the styleable were divided in three categories:

1. Site owner level attributes: Only a site owner would be able to set these attributes.
2. Domain owner level attributes: Only site owner or the owner of the domain to which the style is associated would be able to set these attributes.
3. List owner level attributes: Only site owners, owners of the domain in which the list is present or the owner of the list to which the style is associated would be able to change these attributes.

Mutability: A style can be made immutable. Once set to immutable, any of its styleable attributes can't be changed by anyone and it can't be set to mutable again.

Style Deletion: If a user deletes a style which is parent of another style then that style would only loose its presence in the list of styles visible to the user but it will not be deleted from the database till any of its child is present in the database. As soon as all of its children are deleted from the database then that style will also get deleted.

So what changes does the new design introduces?

Earlier, a style which was supposed to be describing a complete set of attributes continues to do so but now instead of acting as a container for all the styleable attributes, it now acts as a stack of partial set of attributes known as stylets. These stylets contain only a partial set of attributes describing only a handful of settings that the user wants to change in a style. A stylet when applied to a style changes the value of the common attributes of all the previous stylets.

Inheritance: A style can still inherit from another style. The inheriting style(and consequently the mailing list using that style) will reflect any changes made in the parent style.

Viewability and attributes categories: Both styles as well as stylets are having same three level of viewabilities, same as before and attributes are also categorized same as before.

Deletion: Earlier a user was allowed to delete a style even if it is was parent of some another style but now a style can be deleted only if it is neither being used by any mailing list nor by any of the style. Similarly, a stylet can be deleted only if it is not in use by any of the style.

Advantages: Makes modification of style easier since the user can now change a handful of attributes that he wants to without caring for other attributes and also apply the same modification to other styles as well.

Stay connected for more.

Thanks,
Harshit Bansal

Pre Midterm Evaluation

Hi readers,
It has been a long time since my last post, in fact I have missed the deadline for my first progress update. I am sorry about that. It was because my project was lagging behind the timeline due to some design and it became my first priority to get the project back on track and I was so much busy in doing so that I almost forgot I also have to maintain my blog. As all of us know, coding period phase 1 has come to end. It was a great time involving intense discussions, coding, roadblocks as well as brainstorming and now its time for midterm evaluations.

Discussions:

During this period, I had an intense discussion with Barry. In first week of the coding period, I was working to implement the new IStyle and IStyleManager interface according to my proposal. While implementing the new IStyle interface I felt that in order to ensure that the attributes dictionary passed to the __init__ method contains all the styleable attributes(since previously, I was assuming styles to be a complete set of attributes), I will need to have a list of all the styleable attributes. So I decided to go for code refactoring by moving all the styleable attributes to a new interface 'IStyleable' interface which then can be extended by the existing IMailingList interface and the new IStyle interface. I asked Barry for his permission for this refactoring and he agreed but he also said that it would be good if I make styles composable in terms of small set of attributes(stylets) as the current IStyle interface supports. We then had a long discussion on what would be the behaviors of these new stylets and how can these stylets be used to compose styles. Subsequently, the new design was finalized(I will be explaining the new design in my next blog post) and I got to realize that my original design was having a large number of shortcomings that I would have to face while implementing the REST API and the frontend.

Brainstorming:

Now I needed to devise a mechanism such that if an user updates any stylet or style, then, any mailing list using either the updated style or a style composed of the updated stylet will get updated automatically. For accomplishing this task I was having an option to override the __setattr__ method and trigger a stylet/style updated event whenever this method is called but as this would trigger the event for each and every assignment, an instance managing a large number of stylets/styles would come to its knees within seconds.
After wasting some time in devising some way of solving this problem, I cam to know about SqlAlchemy's mapper level events and after reading their documentation, I decided to use them.

Roadblocks:

Unfortunately, during this period I faced two major roadblocks:
1: During the time of discussion, Barry was busy in PyCon 2016 and was not available on IRC for a real time discussion. So I had to contact him via email which was very time consuming and I was not able to make much progress during that week. But I have to say that in-spite of being busy, Barry tried to answer all of my queries as clearly and as fastly as possible(Thanks to him!).
2: When I was testing the new event system I came to know that even after changing a style/stylet the events aren't getting fired. Then after spending some time in trying to figure out the problem, it came to my notice that if I fire a query after changing a style, the events are fired in proper order. I was unable to figure out any possible reason for this kind of behavior. I found a solution to this problem on stackoverflow. Events in sqlalchemy are fired when a flush occurs and a query causes an auto-flush to be triggered which causes the events to be fired. It took about three days to figure it out and caused a lot of frustration.

Overall it was an amazing experience to see my project taking shape and solving those roadblocks was the most interesting part although it caused some frustration as well but I loved it. Keep connected for more.

Thanks,
Harshit Bansal

Proposal Explained

Hi readers,
In this post I will try to explain the project that I will be doing this summer.

Project Title : Preset List Settings Template(aka Styles)
Organization  : GNU Mailman
Mentors          : Abhilash Raj, Barry Warsaw, Terri Oda, Florian Fuchs

Some basics:
What is Mailman?
Mailman is a piece of software that is used for managing mailing lists, newsletters etc. It mainly has four components:
1: Core- It is responsible for the actual delivery of the emails, management of user data(subscriptions, preferences etc.) and for the handling of the requests made by REST API.
2: Postorius- It is official web user interface of the Mailman which helps users(list members, list admins, moderators) in managing their settings(preferences, subscriptions) and the settings of their lists.
3: Mailman Client- It provides the glue code which enables a developer to write its own scripts and front-ends which can communicate with the core without caring for the underlying details and changes. Postorius also uses it for communicating with the core.
4: Hyperkitty- It is official archiver of the Mailman which provides access to the list archives and provides a way to interact with the lists.

What's the purpose of a mailing list?
A mailing list is simply a list of e-mail addresses of people that are interested in the same subject, are members of the same work group, or who are taking class together. When a member of the list sends an email to the group's special email address(the address of the mailing list), the e-mail is broadcast to all of the members of the list automatically. The subscribers can opt either to receive the mail immediately or as digests(a single mail containing all the new mails posted to the list).
At least there can be two type of mailing lists:
1: Announcement lists: These type of lists are used for broadcasting information to a group. The users who subscribe to these lists are eligible for receiving mails from this list but they can't send mails to the list.
2: Discussion lists: These types of lists are meant for sharing ideas, discussing some thing about some particular topic etc. Each member can send message to list distribution to all other members which in turn can then share their ideas, give feedback etc.

What is a 'Style'?
Managing a mailing list may be a tedious task. The list administrators need to manage a number of settings which govern how their list behaves. Mailman refers these set of settings as 'Style'. When we define a new 'Style' we create a new set of settings that can be applied to a mailing list. Mailman uses a database to store the 'settings' of a list. By 'applying a style to a list' we mean to take the 'values' of the attributes(settings) from a style and copy them to the database.

Present Scenario(Or the problem)-
Currently for defining a new style, the users need to create a file containing a python class which implements the 'IStyle' interface. This class will be containing all the necessary details required for defining the new style and the name of this class will be the name of the new style(which has to be unique). This file then needs to be copied to one of the import paths so that mailman can load it. This is process is much more tedious than it seems. Also only the users having shell access to the machine are able to define a new style. Furthermore, managing styles becomes an extremely difficult task. For defining a new style, it becomes mandatory to know Python(as he/she will need to write python code). Not only this, there is a one more big issue that is: At present a style can be applied to a list only when it is created. It means that if we change a style, then, the settings of the mailing lists using that style are not updated automatically. This posses a serious problem in managing the list. For tackling this problem, the Postorius provides a "List Settings Form" which enables a user to change the settings of the list. But that too contains just a handful of the settings and if the user has to take the support of an external web form for managing the list settings then it becomes a serious question on the usefulness of the styles in managing the list settings.

Aims of my project-
My project aims at providing a web interface(as part of Postorius) that will help the users in creating the new styles and managing the existing ones and to make it possible to manage the list settings via modifying a style, i.e., if a user updates some settings in the style then those the corresponding settings of the mailing list using that style should automatically get updated. This will make the whole process of defining a new style and managing a list's settings using it just a matter of few clicks.

That's it for now. I hope that it was not difficult to follow. I will be posting more of the technical details about the project as soon as my semester exams end.

Thanks,
Harshit Bansal

My GSOC journey so far

Hi readers,
This is going to be my first blog post and somewhat longer as well. It is an informal post not related to my GSOC project. In this post I will be focusing mainly on my journey of GSOC. Okay so lets get started- :)

How all of this started? -
All of this started about one and a half year ago during my stay in Kota for my preparation of various competitive exams(Yes, I am a dropper). When I was in Kota I had made it my habit to browse quora and google news daily for fifteen minutes so that I can stay updated with worldly affairs apart from PCM. It was like a normal night, the sky was clear, the stars were twinkling(don't worry not going to write a long story, just setting up the mood), I was browsing quora on my Micromax Q5(my phone earlier) then all of a sudden a question poped in my Quora feed. If I remember correctly, it was like - How should one prepare for "GSOC"? (I was unable to even pronounce the last word correctly!). I had no idea about GSOC. So I googled "GSOC"(from that day I have made it my habit to google any term that I come across, whose meaning I don't know. Never know what the luck has in its pocket for you!). It took me about five to six days to completely understand how the program works and what are its prerequsites. Since before this I had never heard of the term "opensource" so I started to google about it and to my mere surprise it turned out that the browser that I am using for all this is also an open source software. I was surprised to see how vast the open source world is. But since I was on my "mission" to crack some competitive exam so that I could land into a good college with a branch of my choice(CS/IT), I had to forget about all this and to concentrate on my studies. I put GSOC aside and continued my preparation but it was now on my target list. All of this happened even before I started my engineering. ;)

As we know that "Time and Tide waits for none", time passed away sooner then I expected and it was now exam time. After facing a mix of some success and failures and a lot of hardships, I was able to grab a seat in HBTI, Kanpur with a branch of my choice(I.T.)(There is a long story behind it as well that I hope to share some time later). Between all of this, there was a time period of about one and a half when I was completely free and since it was almost a year that I was from programming, the programming enthusiast inside me was desperate to start coding again. It was during this time period that I got to know about "Python". I got "attracted" towards it and started to learn it. Since I already had a good knowledge of C and Java, Python seemed to be a piece of cake and within just one month I got proficient in it. Knowing about my  passion for programming, my parents gifted me a really nice laptop. Now to have some practice of my newly learned language and to have a view of the open source world I started to contribute to Mozilla(GSOC was still not in the picture). While contributing to Mozilla I got to learn about a plethora of new things like Linux, version control, unit testing etc. It was really a nice experience. The people there were really helpful.

About five months of hard work, by the end of December I was now very much comfortable with the terminology and the modus-operandi of the open source world. Earlier, I was having plan to apply for GSOC in my second year of engineering. But being confident enough about my skills I decided to apply for it in my first year. I thought that if I get selected then it will be a big achievement and a dream come true earlier than expected and even if I don't get selected I will be having the "necessary experience" so that I can crack it next year and it really turned out to be a really good decision.
That's it for now. I will be adding more posts sooner so stay connected for more. I hope that it wasn't boring.

Thanks,
Harshit Bansal.