-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unclear error message for concat and IamDataFrame.append and potentially undesired behaviour #707
Comments
Sorry @jkikstra, but I'm afraid that this is expected behavior. pyam is quite "generous" when it comes to additional data dimensions - it automatically assumes that any column named other than model, scenario, ... or a time dimension is an additional relevant identifier (aka "extra-column", see here). This is intended to make it user-friendly when working with e.g. subannual timeslices or additional columns to distinguish between different climate models (this is used in open-scm, I believe) pandas reads the first "index" column as Possible solutions:
|
Okay great! If we expect that this occurs relatively often, given the prevalence of (1.) Indeed a step further would be to add a function to do similar things. Maybe rather (2.) Hmm yes, so concat or append with an option like join_by could be a nice option. But if users are generally still not using extra columns much, that may be lower priority. Especially if you already do (the potentially simpler 0 and 1), users can deal with thing in code without having to touch the data files? |
Short error description:
When using either
pyam.concat()
orIamDataFrame.append()
on two dataframes that come from files that have a column with row indices, I get an unexpected error message, sayingIncompatible timeseries data index dimensions
:Moreover, it is possible to read in both files separately as a pyam.IamDataframe, so I am not entirely sure this current behaviour is desired, as one would expect to be able to concatenate to pyam.IamDataframes.
Deleting column with the indices resolves the issue.
Detail on the data:
The files looks like this:
Note:
Somehow, the first two files merge fine, while the third throws the error.
How to reproduce:
For iiasa colleagues, run the notebook: "I:\kikstra\shape-data\Combine files with pyam.ipynb"
The text was updated successfully, but these errors were encountered: