[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add missing handler for deserializing json value #1587

Merged
merged 7 commits into from Dec 13, 2023

Conversation

tomwojcik
Copy link
Contributor

Fixes #1500

@tomwojcik tomwojcik requested review from a team as code owners June 19, 2023 10:59
@tomwojcik tomwojcik requested a review from tswast June 19, 2023 10:59
@google-cla
Copy link
google-cla bot commented Jun 19, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@product-auto-label product-auto-label bot added size: s Pull request size is small. api: bigquery Issues related to the googleapis/python-bigquery API. labels Jun 19, 2023
@tomwojcik
Copy link
Contributor Author
tomwojcik commented Jul 2, 2023

@tswast bump, please review

@parthea parthea added kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Jul 6, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 6, 2023
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 6, 2023
@tswast tswast added the kokoro:run Add this label to force Kokoro to re-run the tests. label Oct 3, 2023
@yoshi-kokoro yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Oct 3, 2023
Copy link
Contributor
@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

One concern I have is the other direction. If someone supplies a parsed JSON object to an API like insert_rows, do we correctly serialize that?

@Linchin Linchin added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 1, 2023
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 1, 2023
@tswast tswast added the kokoro:run Add this label to force Kokoro to re-run the tests. label Dec 13, 2023
@yoshi-kokoro yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Dec 13, 2023
@tswast
Copy link
Contributor
tswast commented Dec 13, 2023

Looks like everything is working well except for the type annotations:

nox > Running session mypy
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/mypy
nox > python -m pip install -e '.[all]'
nox > python -m pip install mypy==1.6.1
nox > python -m pip install types-protobuf types-python-dateutil types-requests types-setuptools
nox > mypy -p google --show-traceback
google/cloud/bigquery/query.py:472: error: Cannot call function of unknown type  [operator]
google/cloud/bigquery/query.py:629: error: Cannot call function of unknown type  [operator]
google/cloud/bigquery/query.py:778: error: Cannot call function of unknown type  [operator]
Found 3 errors in 1 file (checked 54 source files)
nox > Command mypy -p google --show-traceback failed with exit code 1
nox > Session mypy failed.

@tswast tswast self-assigned this Dec 13, 2023
@tswast tswast added kokoro:run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Dec 13, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 13, 2023
@yoshi-kokoro yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Dec 13, 2023
@tswast tswast merged commit 09017a9 into googleapis:main Dec 13, 2023
18 of 20 checks passed
@tomwojcik tomwojcik deleted the feature/#1500-jsonfield-deserialize branch December 13, 2023 22:48
@MartijnvanElferen
Copy link

This change is causing the following error for insert_rows:

TypeError: _json_from_json() missing 1 required positional argument: 'field'

Is there a recommended fix for this?

@tswast
Copy link
Contributor
tswast commented Dec 15, 2023

Thanks @MartijnvanElferen for the report. I've filed #1756 to investigate further.

@tswast
Copy link
Contributor
tswast commented Dec 15, 2023

As a workaround insert_rows_json should avoid this type conversion code, but you're responsible for putting things in a format the BigQuery REST API understands.

@tswast
Copy link
Contributor
tswast commented Dec 15, 2023

@MartijnvanElferen Fix pending: #1757

Note: this PR will change the behavior of JSON data in insert_rows to call json.dumps, which is consistent with the new behavior of calling json.loads when reading row values. Previously, JSON support was happening via a generic fallback that didn't do any conversion of values passed in.

@MartijnvanElferen
Copy link

Thanks, @tswast. I'll refactor a couple of things to avoid the type conversion. Looking forward to the fix at the same time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. size: s Pull request size is small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Deserializing JSON subfields within structs fails
6 participants