Simplified and reconfigured model building and application

[shankari]

• Use only the first stage
• up the radius to 500m
• no filtering, no cutoffs for the similarity code
• user is valid if they have > 14 trip (no % requirement)
• user input is valid if any entry has been filled out (before all entries had to be filled out)

Improvements from these changes, compared side by side with the old model, are at:
e-mission/e-mission-eval-private-data#28 (comment)

• bonus fix: fix the regression in the tests by handling the corner case of no user inputs.

[shankari]

Just wanted to call out these highlights:

There are still many users for whom the new model does not predict a lot of trips, but it is much much better.

The only users with < 20% trips prediction rate have very few (<50) labeled trips.

[shankari]

Running against the full dataset now; will deploy to staging tomorrow morning when I am not so brain dead.
Don't want to make a mistake and end up deleting all the data.

[shankari]

After running the full intake pipeline against the staging DB from last week for one user, I get

all inferred trips 243
all confirmed trips 243
bin/debug/label_stats.py:40: DeprecationWarning: count is deprecated. Use Collection.count_documents instead.
  print("confirmed trips without inferred labels %s" % (edb.get_analysis_timeseries_db().find({"metadata.key": "analysis/confirmed_trip", "user_id": sel_uuid, "data.inferred_labels": []}).count()))
confirmed trips without inferred labels 205
confirmed trips with expectation 243

[shankari]

The pipeline ran successfully against the staging database from last week on my laptop.
Now getting ready to run on staging!

[shankari]

Found at least a couple of users with:

Found confirmed trip with matching inferred trip, without user labels
Found confirmed trip with matching inferred trip, without user labels
Found confirmed trip with matching inferred trip, without user labels
Found confirmed trip with matching inferred trip, without user labels
Found confirmed trip with matching inferred trip, without user labels
Found confirmed trip with matching inferred trip, without user labels

[shankari]

Resetting the inferences; due to e-mission/e-mission-docs#654
this took a few minutes to delete all the expected trips, at least for one of the users.

# ./e-mission-py.bash bin/debug/reset_partial_label_testing.py -i
Connecting to database URL stage-db
{'n': 5148, 'ok': 1.0}
{'n': 813198, 'ok': 1.0}
{'n': 10297, 'ok': 1.0}
{'n': 5148, 'ok': 1.0}
{'n': 76, 'ok': 1.0}

[shankari]

Resetting the confirmed trips

./e-mission-py.bash bin/debug/reset_partial_label_testing.py -c
Connecting to database URL stage-db
{'n': 0, 'ok': 1.0}
{'n': 5148, 'ok': 1.0}
{'n': 38, 'ok': 1.0}

Why do we have inferred trips for some and not for the others? Too late to debug now...

[shankari]

Re-running the pipeline to generate confirmed trips so we can build the model.

# ./e-mission-py.bash bin/intake_multiprocess.py 3
Connecting to database URL stage-db
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
overpass not configured, falling back to default overleaf.de
transit stops query not configured, falling back to default
...
2021-07-27T14:42:19.965960+00:00**********UUID ae91f6bc-b25d-4d80-93ba-0159ac220901: storing views to cache**********

Next, building the model

# ./e-mission-py.bash emission/analysis/modelling/tour_model_first_only/build_save_model.py > /tmp/build_model_jul_27.log 2>&1
# ls user_labels* | wc -l
19
# grep "doesn't have enough valid trips" /tmp/build_model_jul_27.log  | wc -l
20

Huh! On my laptop, I got

$ ls user_labels_first_round_* | wc -l
      20
$ grep "doesn't have enough valid trips" /tmp/staging_run.log | wc -l
      21

Need to follow up on the discrepancy.

Running the pipeline once so we can fill in the inferred_trip and expected_trip objects. The confirmed_trips can be filled in during the regular run.

# ./e-mission-py.bash bin/intake_multiprocess.py 3
2021-07-27T14:55:19.880112+00:00**********UUID ae91f6bc-b25d-4d80-93ba-0159ac220901: storing views to cache**********

Resetting the confirmed trips so they will be regenerated on the next run.

# ./e-mission-py.bash bin/debug/reset_partial_label_testing.py -c
Connecting to database URL stage-db
{'n': 10296, 'ok': 1.0}
{'n': 5148, 'ok': 1.0}
{'n': 38, 'ok': 1.0}

Spot checking a couple of entries

# ./e-mission-py.bash bin/debug/label_stats.py -u [redacted]
Connecting to database URL stage-db
All inferred trips 176
Inferred trips with inferences 0
All expected trips 0
Expected trips with inferences 0
all inferred trips 176
all confirmed trips 0
confirmed trips with inferred labels 0
confirmed trips without inferred labels 0
confirmed trips with expectation 0

Will double check after the next pipeline run to confirm that the confirmed trips are created.

@GabrielKS you can go ahead and test against staging now!

[shankari]

For the same user, we now have:

(emission) root@a5b7fc6b3e7e:/usr/src/app/e-mission-server# ./e-mission-py.bash bin/debug/label_stats.py -u [redacted]
Connecting to database URL stage-db
All inferred trips 176
Inferred trips with inferences 0
All expected trips 352
Expected trips with inferences 352
all inferred trips 176
all confirmed trips 176
confirmed trips with inferred labels 0
confirmed trips without inferred labels 176
confirmed trips with expectation 176

[shankari]

spot checking several users, most of them don't have confirmed trips with labels.

Checking the aggregate:

>>> import emission.core.get_database as edb
Connecting to database URL stage-db
>>> edb.get_analysis_timeseries_db().count_documents({"metadata.key": "analysis/confirmed_trip", "data.inferred_labels": {"$ne": []}})
1

wha!! there's only inferred label.

Maybe we fixed e-mission/e-mission-docs#654 so now we shouldn't delete the expected trips any more without resetting the pipeline. Let's regenerate everything without the hack and see if it works.

[shankari]

@corinne-hcr Can you please merge this to your branch as well? It should fix the failing test....