Data ingestion/cleaning system using the publish-subscribe design pattern in Python.

Task/Problem Description:

I have the data producer (module [url removed, login to view] and function get_data) that produces two types of football data: play-by-play and tracking. The play-by-play is a dictionary, and I'd like you to do some error checking, cleaning, and reformatting.

- Rename all camel case field names and sub-field (nested) names with underscores ("playStartTimestamp" becomes "play_start_timestamp").

- Replace all camel case event names (values of the elements in the "events" list with field name "event") with underscore notation ("offensiveFormation" to "offensive_formation").

- Check that all elements of the list "events" occur before or at the play end time and after or at the play start time. If not true, log it and remove the event from the list.

- Remove events "playStart", "reviewStatus", "play_submit", and "playType" by checking the value of field "event" in the "events" list.

- When a field or subfield name is an ID, for example "play_id", capitalize "id" to "ID".

- If any fields are not as you expect, log it.

The second type of data is raw tracking location. This comes in the form of a list of lists. Each element in the top-level list corresponds to a single measurement of the location of a single entity (player, referee, object on the field). The sub-list contains the entity identifier, two location coordinates x and y, and a timestamp.

[// ID x y t

[41021, 10.1, 121.111, 1445559932.34],

['pylon-1', 100, 120, 1445559932.39],

[1034151, 100, 120, 1445559933.11],

...

]

Please reformat it as:

{

ID: {'x': [...], 'y': [...], 't': [...]},

...

}

Both data types are prone to problems that I'd like you to try to correct, when possible. If there are cases when you can't clean the data, please log the bad data when the code encounters it.

After performing these transformations, publish the resulting data to two subscribers that simply write the data to JSON files. One subscriber is for play-by-play data and the other subscriber is for tracking data. You may use libraries, but do not use messaging libraries such as zmq.

I will provided you with the code for the data producers along witht JSON files for Play by play and tracking. If you have anu doubts regarding this you can reach me on mail.

Please send me back the code and the screenshots of the output. You can provide this explanation as comments in the code or as a separate document.


Tags: Linux, Python
Bid on this Job!

 

Here are some random jobs:

Advance Building plan available

Residential and commercial...Read Full Description

Build an X11 Blockchain (Dash fork)

This is the specification for the blockchain:Algorithm: X11 Dash CloneSymbol: QStarting Transaction Letter: QTTotal Supply: 21 BillionPre-mined coin: 2.1 BillionTIme frame: 120secBlock Rewards: 100 BlocksBlock Size: 2 MBHalving: 100 MillionWe see the project l...Read Full Description

Write some blog articles

Can you write 500 word blog articles about slippers, flip flops, sandals etc? Then you might be able to help....Read Full Description

URGENT: Writer Needed

We need a talented and multifaceted writer to handle various writing projects for us on an ongoing basis with English Skill.The successful applicant must have a great command of the English language and a writing style that expresses ideas clearly in a tone th...Read Full Description

I'm finding New Freelancer for my project with English skills - 16/05/2018 06:13 EDT

We need a talented and multifaceted writer to handle various writing projects for us on an ongoing basis.The successful applicant must have a great command of the English language and a writing style that expresses ideas clearly in a tone that is relatable. As...Read Full Description

Copywriter needed to write furniture product descriptions

Task: To write new product descriptions (10) for the furniture for an US-based online shop. Products will be listed in the private message. ... Requirements: Volume: 200 words each product - Use of synonyms for the furniture name and type (Example: sofa bed/sl...Read Full Description

I need an After Effects Specialist

My name is Oliver and I have a company Pixx digital. We create business presentations in virtual studios filmed on green screen using pre-made After Effects templates. We need someone who is very proficient in After Effects and can use the template to put toge...Read Full Description

New Logo Design

I'm looking for a graphic designer to create a simple logo for me - the logo will be for an APP concept that will be part of a presentation I am doing. The name of the APP is "Cart" I would ideally like two logos: 1) the name 'Cart' 2)...Read Full Description

Design a Logo - "Big Brain Big Booty"

I need a logo for a blog project!The Name is "Big Brain Big Booty" as in smart girls with a big bum.I want the design to be Pink (similar to #eb77a6), light Grey (e5e5e5, efefef) and White.It can have any combination of a brain, a head, a bottom, gym...Read Full Description

data

I need someone to submit some articles for me....Read Full Description

Articles, Blogs & web content writer needed...

Hi!Looking for some writer who will be qualified on multiple writings. Skilled writer needed basically. I have many more writing projects. I need 8-10 articles or blogs per week. My budget is $7 to $10 for each article or blog.If anyone impress me for his grea...Read Full Description

Article Writing for Freelancers, Web Design & Development with SEO

Project Description Recruitment Details: Articles/Content writing needed on the topics of Freelancer, Web Design & Web Development. Keywords or Article titles will be given, Suggestions are welcome . Now, we’re hiring people who are best at writing Engli...Read Full Description

Short video intro animation from a photo

My client has an image (attached) that he wants turning into an animation for a video intro. It needs to be punchy and professional looking and include the words:Deano Gamer TVSee clients youtube channel to get an idea of the sort of videos it will be going in...Read Full Description

Design a App Logo and Splash

I have a social media app like facebook which India based.I want you to design a logo and splash with the theme IndiaThe Name of the app Is IndiaBook or ibookI would like to check the images to inspire the ideait would be better to use the only letter i instea...Read Full Description

Create the cover for Music Album

We need to create the design for Itunes/Youtube.. cover for Album[login to view URL]Moodboard is attached...Read Full Description

Search the web for something

Search the web for something.....Read Full Description

 

© 2005 - 2018 getFreeLancer.com
705390 contracts/jobs/projects currently available in our database.