Recommender System
CREU Research - by Elizabeth Durflinger

April 14th, 2019

From April 11th to the 13th our team was in Kennesaw, Georgia for NCUR to present our research. It was a nice experience to bond with the team and visit a new place. Our presentation went well, we had about 10-15 people in the audience with some interesting questions posed. Some were focused on the bugs that we briefly mentioned, some on the organization of the project. Though I was not the one speaking, it influenced me to want to speak at the next conference. I am continuing to fix bugs on the web application. I'm unsure that I will be able to finish it. I have 2 other term projects that are due within the next two weeks. I am hoping to at least finish the web app.

April 7th, 2019

During our team meeting we discussed some issues with the app. The Heroku server is slow. I discovered that this was because I was not on the paid tier for Heroku on my backend server. I fixed this so the web app responds quickly now. Dr. Yu and I tried to fix the connections issues with MongoDB but were unable to. I wrote the backend in JavaScript, and there are some threads on Stack Overflow about issues with MongoClient and Node.js. I haven't had much luck with fixing this. Dr. Yu also requested that I set up a test app on an AWS Server so that he can access the app in a way which he is familiar. I have no clue how to set up that environment so that will also take more time. At the moment I have been spending 15ish hours on this project per week. I hope to have these app issues resolved but the conference.

March 31th, 2019

The web application is live! You can login or register here. I have learned so much through building this application. I will be adding error handling and improving the styles of the web app in the coming weeks. Hopefully it is mobile friendly by NCUR on April 11th. I am presenting a poster for this project at Quest at YSU on April 2nd, and at the COFSP Poster Conference at Case Western University on April 7th.

March 24th, 2019

The web application is finished but one bug is preventing it from moving to production. These are issues with cors but it seems like I did it correctly. I'll have to spend some more time figuring out what is wrong. I have another concern, I am keeping my connections open but it doesn't feel right, I think they should close so that the connections don't go over the max - which is 100. I'll try to figure out a way to fix this. Next will be error handling.

March 17th, 2019

Spring break has come to a close. I have finished most of the frontend and tying it to backend. I currently have the registration, login, reset password, and user dashboard working. I also spun up a heroku server for my frontend. I am halfway through the user profile page and I also realized I do not have error handling in my forms. This is not a huge deal and something I will gradually implement later on. I'm super excited, the webapp is nearly completed and ready to be used by test users.

March 10th, 2019

This week has been studying for midterms and dealing with illness. The backend is done and I have started on the frontend. One of the most difficult things for me through this process has been identifying what tools to use. Having not used them before it is difficult to have a firm grasp on how they work and if I will need them. I accidentially installed Node and my React app in the wrong place and messed up my repository so I had to start from scratch. I will be using spring break as an opportunity to finish the frontend, set up heroku, and figure out my DNS/domain stuff - from there the app should be good to go.

March 3rd, 2019

The past two weeks have been rough, I've had health issues that are preventing me from attending my classes and working efficiently. I and half way through writing the deals endpoint and it should be done soon. Initially I was hoping to have finished the webapp by now but there is a lot more that I had to learn that I first anticipated. I really want to be able to start the frontend this week and get at least a page working. Once I get one I should be able to complete the others in a fairly short amount of time.

Feb 26th, 2019

This week some health issues came up and I was unable to work on the webapp for as much time as I would have liked. I have been putting at least ten hours in per week but in order to get this done in time I will need to spend more. The backend of the app is proving to be challenging for me and I'm having to reach out to other members of the programming community for answers to my questions. I have written the endpoints for /register, /login, and /reset and /profile in my backend. I still have to write the endpoints for the /deals page. This includes a Get for the deals, and a Post for the rating. For this I am using MongoClient. My goal is to have the backend done by next week.

Feb 19th, 2019

I have started development on the web app. I have repos in our organization on Github for the frontend and backend. The database is designed so that a user will have an array rated objects containing deal ids and the rating - this will allow us to extract this data a run it through the reccomender algorithm. I have also designed my API and endpoints. I have decided that I will use MongoDB, MongoClient and Express.js for my backend. The frontend will be made in React.js using Material UI, and use Node.js and Axios. Most of my experience is with frontend development and Backbone so I anticipate this taking some time. Hopefully I will be able to complete this within the next couple of weeks.

Feb 12th, 2019

I have met with Dr. Yu and I realized that there has been a misunderstaning in our scope. I will be making a website that displays the deals and then allows the users to rate the deals. They will submit their ratings which I will store in a table in our database. I will then export this data in a csv file to my team members who will run different algorithms on the data to create recommendations. They will give this back to me and I will push this information to the users. I have the data in the database and I will begin development on this website aiming to get it done in the next 3 weeks.

Feb 5th, 2019

Our team has started the EDA and the development on the web app, I am planning on creating the React app, and getting the data into the database. I transfered the data into a CSV file for EDA. I have formulated a design for the app and UX flow. Dr. Yu and I will go over this in our meeting this upcoming Tuesday.

January 29th, 2019

We did not meet this week due to very cold weather and campus being closed. I have made a MongoDB Atlas account and have created a database. I just need to import the data. I have tried repeatedly to get the JSON into the proper format but I feel that I am struggling with little things that are easy to miss. I am learning a lot but feel that I'm not making as much headway as I would like. I have read several articles on webapp design but I dont fully understand this implementation of the algorithm, I think it will just be a python file that runs upon changes to the model but I am not 100% sure.

January 22th, 2019

Out team has come to an agreement to meet every Wednesday at 3:15 PM, Maddie, Alyssa, and Olivia will be working on Exploratory Data Analysis and I will be working on taking our data, shoving it into a database, and bulding the webapp off of it. I'm not familiar with backend processes so I will have to ask Dr. Yu for assistance on this part. The data is in a JSON file that has an array of deals, it is difficult to import this data and preform EDA in this format. I will be trying to get this into a better format for the EDA and importing. Dr. Yu has asked me to formulate a plan for the design of the project.

January 15th, 2019

This week our team did not meet due to scheduling issues. I continued to look into databases and backend development.

January 10th, 2019

Winter Break Post Mortem: A lot of events happened during winter break, I moved out of my parents house, volunteered at CodeMash, a week long programming conference, and researched React the frontend framework. I was extremely busy due to moving and work and was not able to complete much for the research project specifically but I did have the opportunity to do a lot of professional development. At CodeMash I have been able to go to sessions on React, Automated Testing, and female leadership. I have also participatied in the capture the flag challenges which are specific to security but include other topic oriented challenges. I have been learning about nmap and dig, as well as DNS. I am excited to start this Spring 2019 semester and build the webapp for our research project. While I am a bit nervous about the scope of the project (full frontend and backend) being too big I am excited to take on the challenge. Here's to the start of a great semester!

December 25th, 2018

December 7th, 2018

During this weeks meeting Maddie found a conference for us to attend, it is the Undergraduate Research Programs Conference at The Ohio State University, Columbus, OH. We will be writing and submitting our abstract this week in hopes of being accepted. It was decided that two of us will present while the other two answered questions. Our team also decided that we will be writing a publicaton. I hope to get the web app and recommender system done within the next 3 months so that we will have time to write and master our presentation.

November 30th, 2018

During this weeks meeting we discussed plans for the web app and presenting our work at conferences. We hope to find a conference to present our work at next semester. We have collected a lot of the data but we need to put in into the database so that we can start coding the web app. I will be drawing up wireframes for the web app for next week and looking for potential conferences for us to attend.

November 22th, 2018

Happy Thanksgiving! Our team did not meet this week due to the holiday. I am very thankful to be part of this exciting project with this group of awesome and motivated individuals. Wishing everyone a Happy Thanksgiving and safe and fun-filled break.

November 14th, 2018

I was unable to make this week's meeting due to having the stomach flu. I didn't get munch done with week due to illness and catching up with other academic responsibilities. I hope to make a comeback next week and get my tasks done. I have started a trello (task manager) board for this project to help me stay on task and on time, since it is nearing the end of the semester I am struggling to juggle all of my responsibilities as well as self care. This includes cards with due dates and push notifications for writing and completing journal entries as well as task relative to our research. For this upcoming week I plan to use the data set given to us by Dr. Kerns to explore more with the apropri algorithm and write a python script that will fetch data from out provider site.

November 8th, 2018

This week our team discussed the literary review and the topics covered in them. We discussed the concept of Single Value Decomposition and Dr. Kerns explained this to us. I'm having a little trouble understanding these Single Value Decomposition in terms of matrixis, I will look for some videos on these topics to help me understand them better. I feel that we should soon be moving to the collection and storing of data. I have been working with some python scripts and experimenting with data collection. I would like to discuss this at the next meeting.

October 30th, 2018

Today I met with one of our advisors Dr. Yu. We dicussed some of the architecture for the end goal of the project, such as how the frontend of our deals website is going to be organized. It was mentioned that we may use React as our frontend library. I am very excited to use this. Most of my expereince is with Backbone from working at Drund as a Frontend Engineer. React is a newer library that is being used more frequently. Dr. Yu and I also discussed using AWS and MongoDB to handle storing the data and using the database. I will be looking more into React, Node, MongoDB, and AWS in the next week to have a better idea on how we are planning to structure the project. For this part of the project we will be collecting data and using it to create our own deals webstie with test users. For this we will need (in the least) login/auth, filters, and likes functionality on the frontend. We will probably add more definite functionality as we get closer to creating the frontend for our deals site. I will be looking into these frameworks and the readings that Dr. Kerns sent us in preparation for next week.

October 24th, 2018

Today our team met and I gave a talk on GitHub and how I propose we use it. This covered the basics of version control and organization. Once everyone was on the same page with GitHub, Dr. Yu announced that we have formally been granted permission to collect data from a deals website and will start data collection in November. This has been blocked for several weeks so it was a relief that the admins of the deals website granted us permissions. Dr. Yu then went on to show us how we will be using the site's API to collect data. We decided that we will not be meeting next week as a group due to Halloween. I will be meeting with Dr. Yu on Tuesday to learn more about the scraping process and how we will be organizing our data. He will be showing me other examples of data collection and manipulation. Dr. Kerns will be sendin the teams articles for literary review which we will read in preparation for our next meeting on November 7th.

October 17th, 2018

This week during our meeting, the team and mentors discussed the progress we made with our programs. Dr. Kerns made a program that worked well and displayed ranked support and confidence as well as association rules. Our team also wanted to be able to access Dr. Kern's code as an example so I helped with setting us all up with a github organization YSU-CREU-2018. Throuhgout the meeting I realized that it would be necessary to host a GitHub workshop to get everyone up to speed on using GitHub and the benefits of version control. Maddie reserved us a room for next week where I will give this presentation. For this week I will prepare a talk on GitHub and look into how to organize our practice programs that we have been working on for the past couple of weeks.
This week, I was able to meet with a software developer through Code Youngstown that was able to help me out with my environment issues on my computer. I will be able to run Python files on my computer without issue.

October 10th, 2018

Over the weekend I was able to purchase a new Macbook Pro. Configuring it posed some issues. I am no longer able to run the program in the terminal. I hope to solve this problem by our next meeting on October 17th. I am 90% sure that I am missing a installation. I plan to confide in a professional python dev in the Youngstown area to help me solve this problem. Once I am able to iterate I will be able to see if the changes I made to my program work.
Our meeting this week went well, we have all made some for of progress on the program. Most of our issues that we are comming across are technical IDE and dependancy issues. I will be looking into our teams techincal problems and hopefully help fix these issues by our next meeting.

October 3rd, 2018

Note: submission late due to computer crashing

Our meeting this week consisted of our members going over our progress with the association rules program. Mine was running but due to my computer breaking I was unable to make progress on getting the data to be displayed correctly. Since I update this blog from editing my personal website (not a 3rd party Wordpress etc) I am updating a week late. I did some more reading into implementing the apriori algorithm and data science. Our assignemt remains to work on the program and get it to display in a better way. I hoped to purchase a new laptop so that I could continue working on the project as well as work and school.

September 24th, 2018

At this weeks meeting (September 19th), our team met with Dr. Kerns to review our progress. We discussed that we will not be using lift to calculate the association rules and that the next step of the project will be to right programs in Python to create association rules for the small data set that we have been given as specified in my last post. I researched the Apriori algorthim how to implement it. From there I created a program that reads from a csv file and creates a set of association rules based on the given information. I need to work on displaying this information in a more user friendly way.
This is how it is currently displayed:

The goal is to have this displayed by association rule in a readable format.

September 15th, 2018

This post pertains to rating matricies and calculating their lift. Lift is defined as how likely one deal will be liked while another is liked. A lift of 1 implies no association between the two deals and a lift greater than one implies that deal one is likely to be bought with deal two. Lift{A=>E} = (Support{A,E})/(Support{A}*Support{E}).

So Lift{A=>E} = (2/4)/(2/4 * 3/4) = (2/4)/(6/4) = 1/3

This means that that E is not likely to be liked when A is liked.

Example Ratings Matrix
Alice 1 0 0 0 0
User 1 1 0 0 0 1
User 2 1 0 0 1 1
User 3 0 1 0 0 1
User 4 0 1 0 1 0

September 12th, 2018

This evening, our CREU Research team met to discuss our goals and milestones. During this meeting we reviewed the association rules, support, and confidence. Dr. Yu assisted us with understanding the concepts of support and confidence and how to calculate the values of each base upon the ratings matrix. Moving on in familiarizing ourselves with reccommender systems, our assignment for this upcoming week is to calculate the lift based on the ratings matrix provided and to research the Apriori Algorithm.

September 7th, 2018

On September 5th, our CREU Research team met to discuss our goals and milestones. Dr. Lucy Kerns invited us all to a Slack workspace for our team, and we joined the main CREU Slack as well. All of our members had set up the basis for their blogs and read over the suggested readings. We discussed association rules, support, confidence, and lift. In order to grasp a more solid understanding of these concepts, our assignment was to calcuate a list of asscociation rules and their corresponding support and confidence values based on a binary ratings matrix. We are to discuss our findings with the group at the next meeting on September 12th.

This next section pertains to rating matricies and calculating their association rules as well as the corresponding confidence and support values. For example, the support of the statement column A => E would be 2/4 and the condidence would be 2/2. For the statement B => D the support would be 1/4 and the confidence would be 1/2.

Example Ratings Matrix
Alice 1 0 0 0 0
User 1 1 0 0 0 1
User 2 1 0 0 1 1
User 3 0 1 0 0 1
User 4 0 1 0 1 0

August 29th, 2018

Our CREU Team met for our first meeting on August 29th, 2018. This includes Dr. Feng Yu, Dr. Lucy Kerns, the faculty advisors, as well as Olivia, Allyssa, Maddie, and myself. In this meeting we discussed our strategies for approaching the requirements of the project and came to the following timeline. In September our team will focus on familiarizing ourselves with the reccomender system, association rules, and matricies. Our faculty advisors will be providing us with reading materials to guide us in our learning. In October through December we will be collecting data, which will be Dr. Yu's field of expertise and keeping a blog of our progress. Next semester will be the bulk of the programming section of this research where we will be making a frontend interface for a deals site and making a reccomender system.

It has been decided that our team will have weekly meetings, every Wednesday from 4:00PM - 5:00PM in Lincoln Building on YSU's campus. In preparation for our next meeting our team should have created a blog and read over the reading selections provided to us by our faculty advisors.