Facebook has submitted a patent to use algorithms based on tagging, hashtags and face recognition technology to predict household features of a user, e.g., household size and demographic composition.

The patent, filed with the USPTO, describes an online system that predicts features of a user based on image data of the user, e.g., profile photos, photos posted by the user and photos posted by other users socially connected with the user, and textual data in the user's profile that suggests relationships among individuals shown in the image data of the user.

The online system applies one or more models trained using deep learning techniques to generate the predictions. For example, a trained image analysis model identifies each individual depicted in the photos of the user; a trained text analysis model derive household member relationship information from the user's profile data and tags associated with the photos. The online system uses the predictions to build more information about the user and his/her household in the online system, and provide improved and targeted content delivery to the user and the user's household.

They will be using photos from both Instagram and Facebook to identify people and then connect them with the users who have been tagged or posted the photo. The tech will cross-reference that information with historical data of past posting, messaging, as well as hashtags used to determine the relation. The most important factor in the determination of household data will be the use of a shared IP device in the household from multiple user sign-ins.

The patent was submitted back in May of last year, however, was made public this last week.

With the advent of online systems, such as social networking systems, content providers have increasingly relied on the online systems to effectively deliver content to their target users in target households. Don't forget that Facebook is not charging you for its services, meaning that you are actually the product. The company has to make money somehow. Facebook is already tracking what you’ve bought online, where you live, your age, your job status, and more to determine what your financial status is and provide that to advertisers. The social network is also sees things that you have finally nor posted online -- a status change that you decided not to post shows Facebook what you were thinking. In addition, by signing in to other apps using yout Facebook credentials make everything easier, but it’s just another way for Facebook to see what you’re doing when you’re off Facebook.

Some months ago, Facebook also launched the "Facebook Portal" which is similar to the Amazon Echo video chatting system. The company says they will be using this information for ad targeting. Not only will members of the household be served the same ads, but it also lets those advertising see how well their ads are doing across household performance. This means that they can tell how often their ads are seen and resulted in at least one conversion from a household member, who is likely to be the decision maker in the household, the average household income, and whether it is a family or roommate situation which will determine how and what ads are targeted to them.