Data ingestion/cleaning system using the publish-subscribe design pattern in Python.

Task/Problem Description:

I have the data producer (module [url removed, login to view] and function get_data) that produces two types of football data: play-by-play and tracking. The play-by-play is a dictionary, and I'd like you to do some error checking, cleaning, and reformatting.

- Rename all camel case field names and sub-field (nested) names with underscores ("playStartTimestamp" becomes "play_start_timestamp").

- Replace all camel case event names (values of the elements in the "events" list with field name "event") with underscore notation ("offensiveFormation" to "offensive_formation").

- Check that all elements of the list "events" occur before or at the play end time and after or at the play start time. If not true, log it and remove the event from the list.

- Remove events "playStart", "reviewStatus", "play_submit", and "playType" by checking the value of field "event" in the "events" list.

- When a field or subfield name is an ID, for example "play_id", capitalize "id" to "ID".

- If any fields are not as you expect, log it.

The second type of data is raw tracking location. This comes in the form of a list of lists. Each element in the top-level list corresponds to a single measurement of the location of a single entity (player, referee, object on the field). The sub-list contains the entity identifier, two location coordinates x and y, and a timestamp.

[// ID x y t

[41021, 10.1, 121.111, 1445559932.34],

['pylon-1', 100, 120, 1445559932.39],

[1034151, 100, 120, 1445559933.11],

...

]

Please reformat it as:

{

ID: {'x': [...], 'y': [...], 't': [...]},

...

}

Both data types are prone to problems that I'd like you to try to correct, when possible. If there are cases when you can't clean the data, please log the bad data when the code encounters it.

After performing these transformations, publish the resulting data to two subscribers that simply write the data to JSON files. One subscriber is for play-by-play data and the other subscriber is for tracking data. You may use libraries, but do not use messaging libraries such as zmq.

I will provided you with the code for the data producers along witht JSON files for Play by play and tracking. If you have anu doubts regarding this you can reach me on mail.

Please send me back the code and the screenshots of the output. You can provide this explanation as comments in the code or as a separate document.


Tags: Linux, Python
Bid on this Job!

 

Here are some random jobs:

Investigar sobre tema etico

Hacer una investigacion y responder los puntos sobre el dilema etico: "¿Puede estar justificada la tortura en algunos casos?a. INDICE PROVISIONALb. ORDENAMIENTO LÓGICO DE LA INFORMACIÓN RECOPILADA PARA DESARROLLAR ELCONTENIDO TEÓRICO :1. FUNDAMENTOS T...Read Full Description

Ongoing Work - Detailed Step-By-Step Instruction Writing Microsoft 365 Applications

LOOKING FOR QUALITY WRITERS THAT ARE A FITSERIOUS INQUIRIES ONLY - PLEASE READ AND FOLLOW INSTRUCTIONS COMPLETELYWe have a team of freelancers working on a long term project. We are looking for a few more individuals who are the right fit to join the effort.R...Read Full Description

Modification in SQL/ Error resolve in Website

Some modification in SQL regarding change in shift timing in existing Software(VB/ASP) , and resolving the receipt sending problem in existing website created in ASP.net...Read Full Description

Write some software

I need you to develop some software for me. I would like this software to be developed for Windows using .NET. [url removed, login to view]Please refer above link.I want above mentioned software in marathi.All functionality n reports must be in marathi....Read Full Description

Dashboard creation

I use recurring appointments in outlook. Need Create visual dashboard to add recurring Appts and so none duplicate time or day in future...Read Full Description

3D Product Evaluation

This project has two parts:1. You will be provided access to existing 3D models and products - you will review them noting what it will take to change them/ update them to updated versioins- once your evaluation is complete you will provide a detailed write u...Read Full Description

WordPress Amazon Affiliate Store

Hi there,We are looking to set up an Amazon Affiliate Store for our WordPress site and is looking for someone with the following skills:- Excellent knowledge of Amazon and how its affiliate program works- Able to search, select and upload the best products to ...Read Full Description

SOFTWARE TOOL

Functionalities:1. website which is responsive on a tablet to punch the water meter readings from each individual property.[url removed, login to view] website should be compatible to upload the existing [url removed, login to view] with the addresses in it so...Read Full Description

Hire a MySQL Developer

Hi there,We are looking to set up an Amazon Affiliate Store for our WordPress site and is looking for someone with the following skills:- Excellent knowledge of Amazon and how its affiliate program works- Able to search, select and upload the best products to ...Read Full Description

Fix usps shipping woocommerce

I need the usps shipping fixed asap on my WP woo commerce. We are down till this is fixed....Read Full Description

Build me a bilingual Megento website

* magento website* 60 pages in English / 60 pages in Traditional Chinese* Design of main pages and translation will be provided * Wire Framing and HTML design* site is 100% responsive and ensure compatibility with all the devices including mobile phones/ iPa...Read Full Description

Dongle huawei e153 calls

I need someone who can configure 1 dongle huawei e153 in centos to use it in chan_dongle on asterisk to make outgoing and receive calls. asterisk is already working with huawei e173 dongle....Read Full Description

Build me a HTML email signature

Looking for 2x pixel perfect HTML email signatures to be built- Must be compatible with retina displays and various devices- They need to be installed in Mac Mail Version 11.2- The "—M" needs to be an image- Phone numbers need to be clickable- Addr...Read Full Description

Need my homepage mobile optomised

Im building this site [url removed, login to view] it looks fine on desktop, but ive noticed when you look at it on mobile, its all displaced and the text is all over the place. Im really just needing someone to go in and fine tune it a bit, Ive also attached ...Read Full Description

LOGOTIPO AFIN.CO

Logotipo desenvolvido para empresa do ramo de Administração Financeira Colaborativa...Read Full Description

Create a company logo

We are a consultancy company. Looking for a bold, elegant and powerful statement logo. with the following:ERA Evolve Recruitment Australia A new ERA in recruitment Please show previous work examples, we would like this logo to be completed in different formats...Read Full Description

 

© 2005 - 2018 getFreeLancer.com
422243 contracts/jobs/projects currently available in our database.