Usbrth

What can I do if neighbor is blocking my solar panels intentionally?

Semisimplicity of the category of coherent sheaves?

High Q peak in frequency response means what in time domain?

Difference between "generating set" and free product?

Who or what is the being for whom Being is a question for Heidegger?

What was the last x86 CPU that did not have the x87 floating-point unit built in?

Can smartphones with the same camera sensor have different image quality?

The variadic template constructor of my class cannot modify my class members, why is that so?

What's the point in a preamp?

system() function string length limit

Did God make two great lights or did He make the great light two?

Take groceries in checked luggage

How to test the equality of two Pearson correlation coefficients computed from the same sample?

Why can't wing-mounted spoilers be used to steepen approaches?

What aspect of planet Earth must be changed to prevent the industrial revolution?

Working through the single responsibility principle (SRP) in Python when calls are expensive

What LEGO pieces have "real-world" functionality?

Mortgage adviser recommends a longer term than necessary combined with overpayments

Would an alien lifeform be able to achieve space travel if lacking in vision?

Can the DM override racial traits?

Did the new image of black hole confirm the general theory of relativity?

Do working physicists consider Newtonian mechanics to be "falsified"?

He got a vote 80% that of Emmanuel Macron’s

Was credit for the black hole image misattributed?

How to handle columns with categorical data and many unique values

4

1

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

$endgroup$

add a comment | 

4

1

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

$endgroup$

add a comment | 

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

$endgroup$

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

asked Apr 8 at 11:04

dungeondungeon

394

asked Apr 8 at 11:04

dungeondungeon

394

1 Answer
1

active

oldest

votes

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314

$endgroup$

add a comment | 

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314

$endgroup$

add a comment | 

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314

$endgroup$

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5941314