Importing a community curated list leads to an error

[panoskpv]

Tandoor Version

Latest

Setup

Docker / Docker-Compose

Reverse Proxy

No reverse proxy

Other

No response

Bug description

When I try to import a community list in greek (el) I get the following error 'There was an error creating a resource!' and nothing is added. When I try to import english version of this list I get the same error, foods, categories etc. are added but foods are added without nutrion data.

Relevant logs

No response

[vabene1111]

can you please post some logs from your container so that i can see whats breaking for you?

[panoskpv]

After your comment i checked the logs and i got dublicate slug error. I removed all added items and for those i wanted to keep i just deleted the open slug field in admin settings. Now i get the below:

{'selected_version': 'el', 'selected_datatypes': ['food', 'unit', 'category', 'property', 'store', 'conversion'], 'update_existing': True, 'use_metric': True}

Internal Server Error: /api-import-open-data/

Traceback (most recent call last):

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _commit

return self.connection.commit()

psycopg2.errors.ForeignKeyViolation: insert or update on table "cookbook_foodproperty" violates foreign key constraint "cookbook_food_properties_food_id_4c533fef_fk_cookbook_food_id"

DETAIL: Key (food_id)=(211) is not present in table "cookbook_food".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/opt/recipes/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner

response = get_response(request)

File "/opt/recipes/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response

response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/opt/recipes/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view

return view_func(*args, **kwargs)

File "/opt/recipes/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view

return self.dispatch(request, *args, **kwargs)

File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch

response = self.handle_exception(exc)

File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception

self.raise_uncaught_exception(exc)

File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception

raise exc

File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch

response = handler(request, *args, **kwargs)

File "/opt/recipes/cookbook/views/api.py", line 1455, in post

response_obj['food'] = len(data_importer.import_food())

File "/opt/recipes/cookbook/helper/open_data_importer.py", line 193, in import_food

FoodProperty.objects.bulk_create(property_food_relation_list, ignore_conflicts=True, unique_fields=('food_id', 'property_id',))

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method

return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/models/query.py", line 796, in bulk_create

with transaction.atomic(using=self.db, savepoint=False):

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/transaction.py", line 262, in exit

connection.commit()

File "/opt/recipes/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner

return func(*args, **kwargs)

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 330, in commit

self._commit()

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 305, in _commit

with self.wrap_database_errors:

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in exit

raise dj_exc_value.with_traceback(traceback) from exc_value

File "/opt/recipes/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _commit

return self.connection.commit()

django.db.utils.IntegrityError: insert or update on table "cookbook_foodproperty" violates foreign key constraint "cookbook_food_properties_food_id_4c533fef_fk_cookbook_food_id"

DETAIL: Key (food_id)=(211) is not present in table "cookbook_food".

[ozokyay]

I just set up an instance with the latest version and ran into a similar issue. I haven't tried today's build (pulling images takes a loooong time on my NanoPi), but I don't see any recent commits to open_data_importer.py, so this might help:

stdout: {'selected_version': 'de', 'selected_datatypes': ['food', 'unit', 'category', 'property', 'store', 'conversion'], 'update_existing': True, 'use_metric': True}
stderr: Internal Server Error: /api-import-open-data/
stderr: Traceback (most recent call last):
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
stderr:     response = get_response(request)
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
stderr:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view
stderr:     return view_func(*args, **kwargs)
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
stderr:     return self.dispatch(request, *args, **kwargs)
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
stderr:     response = self.handle_exception(exc)
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
stderr:     self.raise_uncaught_exception(exc)
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
stderr:     raise exc
stderr:   File "/opt/recipes/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
stderr:     response = handler(request, *args, **kwargs)
stderr:   File "/opt/recipes/cookbook/views/api.py", line 1483, in post
stderr:     response_obj['food'] = len(data_importer.import_food())
stderr:   File "/opt/recipes/cookbook/helper/open_data_importer.py", line 135, in import_food
stderr:     'supermarket_category_id': self.slug_id_cache['category'][self.data[datatype][k]['store_category']],
stderr: KeyError: 'category-dry-goods'

[vabene1111]

thanks, i recently did not have much time for this but need to take another round now that there are quite a few objects added to the repository.

[Kixel]

Hi, i have the same issue on this version:
Current Version: 1.5.4
Ref: c78b7a69288179202a2a054b19f1de5a2790f4dd
with the "de" dataset. How can i get you some logs for this?
I'm running tandoor via a stack (docker compose) in portainer with the reommended docker compose file. It's a quite fresh install (2 days old iirc), running on Lubuntu 22.04LTS.

[AxaliaN]

Yeah same here, while importing any locale's data set.