Hub documentation
Combine conjuntos de dados e exporte
Combine conjuntos de dados e exporte
Nesta seรงรฃo, demonstraremos como combinar dois conjuntos de dados e exportar o resultado. O primeiro conjunto de dados estรก no formato CSV e o segundo estรก no formato Parquet. Vamos comeรงar examinando nossos conjuntos de dados:
O primeiro serรก TheFusion21/PokemonCards:
FROM 'hf://datasets/TheFusion21/PokemonCards/train.csv' LIMIT 3;
โโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโโโโโโโโโโ
โ id โ image_url โ caption โ name โ hp โ set_name โ
โ varchar โ varchar โ varchar โ varchar โ int64 โ varchar โ
โโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโโโโโโโโโค
โ pl3-1 โ https://images.pokโฆ โ A Basic, SP Pokemon Card of type Darkness with the title Absol G and 70 HP of rarity Rare Holo from the set Supreme Victors. It has โฆ โ Absol G โ 70 โ Supreme Victors โ
โ ex12-1 โ https://images.pokโฆ โ A Stage 1 Pokemon Card of type Colorless with the title Aerodactyl and 70 HP of rarity Rare Holo evolved from Mysterious Fossil from โฆ โ Aerodactyl โ 70 โ Legend Maker โ
โ xy5-1 โ https://images.pokโฆ โ A Basic Pokemon Card of type Grass with the title Weedle and 50 HP of rarity Common from the set Primal Clash and the flavor text: Itโฆ โ Weedle โ 50 โ Primal Clash โ
โโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโโโโโโโโE o segundo serรก wanghaofan/pokemon-wiki-captions:
FROM 'hf://datasets/wanghaofan/pokemon-wiki-captions/data/*.parquet' LIMIT 3;
โโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ image โ name_en โ name_zh โ text_en โ text_zh โ
โ struct(bytes blob,โฆ โ varchar โ varchar โ varchar โ varchar โ
โโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ {'bytes': \x89PNG\โฆ โ abomasnow โ ๆด้ช็ โ Grass attributes,Blizzard King standing on two feet, with โฆ โ ่ๅฑๆง๏ผๅ่็ซ็ซ็ๆด้ช็๏ผๅ
จ่บซ็ฝ่ฒ็็ปๆฏ๏ผๆทก็ดซ่ฒ็็ผ็๏ผๅ ็ผ้ฟๆก่ฃ
็ๆฏ็ฎ็็ๅฎ็ๅดๅทด โ
โ {'bytes': \x89PNG\โฆ โ abra โ ๅฏ่ฅฟ โ Super power attributes, the whole body is yellow, the headโฆ โ ่ถ
่ฝๅๅฑๆง๏ผ้ไฝ้ป่ฒ๏ผๅคด้จๅคๅฝข็ฑปไผผ็็ธ๏ผๅฐๅฐ้ผปๅญ๏ผๆๅ่ไธ้ฝๆไธไธชๆๅคด๏ผ้ฟๅฐพๅทดๆซ็ซฏๅธฆ็ไธไธช่ค่ฒๅ็ฏ โ
โ {'bytes': \x89PNG\โฆ โ absol โ ้ฟๅๆขญ้ฒ โ Evil attribute, with white hair, blue-gray part without haโฆ โ ๆถๅฑๆง๏ผๆ็ฝ่ฒๆฏๅ๏ผๆฒกๆฏๅ็้จๅๆฏ่็ฐ่ฒ๏ผๅคดๅณ่พน็ฑปไผผๅผ็่ง๏ผ็บข่ฒ็ผ็ โ
โโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Agora, vamos tentar combinar esses dois conjuntos de dados unindo-os pela coluna name:
SELECT a.image_url
, a.caption AS card_caption
, a.name
, a.hp
, b.text_en as wiki_caption
FROM 'hf://datasets/TheFusion21/PokemonCards/train.csv' a
JOIN 'hf://datasets/wanghaofan/pokemon-wiki-captions/data/*.parquet' b
ON LOWER(a.name) = b.name_en
LIMIT 3;
โโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ image_url โ card_caption โ name โ hp โ wiki_caption โ
โ varchar โ varchar โ varchar โ int64 โ varchar โ
โโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ https://images.pokโฆ โ A Stage 1 Pokemon โฆ โ Aerodactyl โ 70 โ A Pokรฉmon with rock attributes, gray body, blue pupils, purple inner wings, two sharp claws on the wings, jagged teeth, and an arrow-like โฆ โ
โ https://images.pokโฆ โ A Basic Pokemon Caโฆ โ Weedle โ 50 โ Insect-like, caterpillar-like in appearance, with a khaki-yellow body, seven pairs of pink gastropods, a pink nose, a sharp poisonous needโฆ โ
โ https://images.pokโฆ โ A Basic Pokemon Caโฆ โ Caterpie โ 50 โ Insect attributes, caterpillar appearance, green back, white abdomen, Y-shaped red antennae on the head, yellow spindle-shaped tail, two pโฆ โ
โโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Podemos exportar o resultado para um arquivo Parquet usando o comando COPY:
COPY (SELECT a.image_url
, a.caption AS card_caption
, a.name
, a.hp
, b.text_en as wiki_caption
FROM 'hf://datasets/TheFusion21/PokemonCards/train.csv' a
JOIN 'hf://datasets/wanghaofan/pokemon-wiki-captions/data/*.parquet' b
ON LOWER(a.name) = b.name_en)
TO 'output.parquet' (FORMAT PARQUET);Vamos validar o novo arquivo Parquet:
SELECT COUNT(*) FROM 'output.parquet';
โโโโโโโโโโโโโโโโ
โ count_star() โ
โ int64 โ
โโโโโโโโโโโโโโโโค
โ 9460 โ
โโโโโโโโโโโโโโโโ
Vocรช tambรฉm pode exportar para os formatos CSV, Excel e JSON.
Por fim, vamos enviar o dataset resultante para o Hub. Vocรช pode usar a interface do Hub, a biblioteca de cliente huggingface_hub e outros para enviar seu arquivo Parquet, veja mais informaรงรตes aqui.
E รฉ isso! Vocรช combinou com sucesso dois conjuntos de dados, exportou o resultado e o enviou para o Hugging Face Hub.
Update on GitHub