Multimodal knowledge consumption adaptation through hybrid knowledge representation

12260,345

16/666747

October 29, 2019

A system and a method of generating and using a multimodal knowledge representation within an artificial intelligence computing system may include prompting a selection of nodes in a graphical visualization; defining an adaptation strategy based; integrating an adaptation strategy representation with the selection of nodes to generate a multimodal knowledge representation; traversing the selection of nodes in response to a user query; performing an adaptation on the selection of nodes; and updating the knowledge representation based on user feedback to a result generated in response to the user query.

International Business Machines Corporation (Armonk, NY, US)

International Business Machines Corporation (Armonk, NY, US)

1. A method of generating and using a multimodal knowledge representation within an artificial intelligence computing system, the method comprising: prompting, via one or more processors, a selection of nodes in a graphical visualization; defining an adaptation strategy comprising a user profile component stored in a memory coupled to the one or more processors; integrating, via the one or more processors, an adaptation strategy representation with the selection of nodes to generate a multimodal knowledge representation; traversing the selection of nodes in response to a user query to register a traversal result; performing an adaptation on the selection of nodes based on the traversal result; displaying, via a display device, a result generated in response to the user query; receiving user feedback on the result generated in response to the user query, the user feedback indicating a user profession, wherein a meaning applied to one or more terms by the user profession is different than an alternative meaning applied to the one or more terms by a different profession; and in response to the user feedback, updating, via the one or more processors, the adaptation strategy to reflect the meaning applied to the one or more terms by the user profession; performing an updated adaption on at least one selection of nodes based on the updated adaptation strategy, wherein the updated adaptation causes at least one node unrelated to the meaning applied to the one or more terms by the user profession to be removed from the multimodal knowledge representation, and the at least one node that is removed from the multimodal knowledge representation is related to the alternative meaning applied to the one or more terms by the different profession.

2. The method of claim 1 , further comprising applying reasoning and calculation to the selection of nodes.

3. The method of claim 1 , further comprising presenting the graphical visualization and a graphical visualization to a user.

4. The method of claim 1 , further comprising receiving an adaptation strategy from user input.

5. The method of claim 1 , further comprising outputting the result to a user.

6. The method of claim 1 , wherein updating the knowledge representation includes using knowledge endpoints.

7. The method of claim 1 , wherein a node of the selection of nodes represents at least one of: a concept, a content, and a software artifact.

8. The method of claim 1 , defining the adaptation strategy by specifying at least one of: a moment an adaptation may occur, a rule of the adaptation should be performed, a user profile parameter, a platform characteristic, an environmental detail, and a reasoning and calculation result.

9. The method of claim 1 , further comprising parsing a visual form.

10. An apparatus comprising: a knowledge dashboard to prompt, via one or more processors, a selection of nodes in a graphical visualization; an adaptation strategy definition interface adapted to receive an adaptation definition, wherein the adaptation definition comprises a user profile component stored in a memory coupled to the one or more processors; an adaptation injection module to integrate, via the one or more processors, an adaptation strategy representation with the selection of nodes to generate a multimodal knowledge representation; a query engine to traverse the selection of nodes in response to a user query to register a traversal result; an adaptation agent to perform an adaptation on the selection of nodes based on the traversal result, wherein a display device displays a result generated in response to the user query; the adaptation strategy definition interface further adapted to receive user feedback on the result generated in response to the user query, the user feedback indicating a user profession, wherein a meaning applied to one or more terms by the user profession is different than an alternative meaning applied to the one or more terms by a different profession; and the adaptation injection module further adapted to, in response to the user feedback, update, via the one or more processors, the adaptation strategy to reflect the meaning applied to the one or more terms by the user profession; the adaptation agent further adapted to perform an updated adaption on at least one selection of nodes based on the updated adaptation strategy, wherein the updated adaptation causes at least one node unrelated to the meaning applied to the one or more terms by the user profession to be removed from the multimodal knowledge representation, and the at least one node that is removed from the multimodal knowledge representation is related to the alternative meaning applied to the one or more terms by the different profession.

11. The apparatus of claim 10 , further comprising further comprising a reasoning module to apply reasoning and calculation to the selection of nodes.

12. The apparatus of claim 10 , wherein the knowledge dashboard presents the graphical visualization and a graphical visualization to a user.

13. The apparatus of claim 10 , wherein the knowledge dashboard receives an adaptation strategy from user input.

14. The apparatus of claim 10 , wherein the knowledge dashboard outputs the result to a user.

15. The apparatus of claim 10 , wherein updating the knowledge representation includes using knowledge endpoints.

16. The apparatus of claim 10 , wherein a node of the selection of nodes represents at least one of: a concept, a content, and a software artifact.

17. The apparatus of claim 10 , wherein the adaptation strategy is defined by at least one of: a moment an adaptation may occur, a rule of the adaptation should be performed, a user profile parameter, a platform characteristic, an environmental detail, and a reasoning and calculation result.

18. A non-transitory computer-readable storage medium for generating and using a multimodal knowledge representation within an artificial intelligence computing system, the non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to: prompt, via the one or more computer processors, a selection of nodes in a graphical visualization; define an adaptation strategy comprising a user profile component stored in a memory coupled to the one or more computer processors; integrate, via the one or more computer processors, an adaptation strategy representation with the selection of nodes to generate a multimodal knowledge representation; traverse the selection of nodes in response to a user query to register a traversal result; perform an adaptation on the selection of nodes based on the traversal result; cause display, via a display device, of a result generated in response to the user query; receive user feedback on the result generated in response to the user query, the user feedback indicating a user profession, wherein a meaning applied to one or more terms by the user profession is different than an alternative meaning applied to the one or more terms by a different profession; and in response to the user feedback, update, via the one or more computer processors, the adaptation strategy to reflect the meaning applied to the one or more terms by the user profession; perform an updated adaption on at least one selection of nodes based on the updated adaptation strategy, wherein the updated adaptation causes at least one node unrelated to the meaning applied to the one or more terms by the user profession to be removed from the multimodal knowledge representation, and the at least one node that is removed from the multimodal knowledge representation is related to the alternative meaning applied to the one or more terms by the different profession.

19. The computer program product of claim 18 , wherein the adaptation strategy is defined by at least one of: a moment an adaptation may occur, a rule of the adaptation should be performed, a user profile parameter, a platform characteristic, an environmental detail, and a reasoning and calculation result.

20. The computer program product of claim 18 , wherein the computer-readable program code is executable by the one or more computer processors to output the result to a user.

6064971 May 2000 Hartnett
7043474 May 2006 Mojsilovic et al.
7853864 December 2010 Ro et al.
8918431 December 2014 Mark et al.
9552339 January 2017 Kessenger et al.
9788349 October 2017 Levien et al.
9946739 April 2018 Haddock
10078651 September 2018 Kelly et al.
2005/0010605 January 2005 Conrad
2005/0265607 December 2005 Chang
2006/0290709 December 2006 Omi et al.
2008/0147515 June 2008 Abraham et al.
2010/0100546 April 2010 Kohler
2011/0264649 October 2011 Hsiao et al.
2015/0100943 April 2015 Gabel
2017/0201562 July 2017 Moon et al.
2017/0212930 July 2017 Carter
2018/0102062 April 2018 Livni
2018/0239959 August 2018 Bui
2019/0065589 February 2019 Wen et al.
2019/0236394 August 2019 Price et al.
2019/0361795 November 2019 Cole
2019/0373330 December 2019 Bloch
2020/0215695 July 2020 Cristache
2021/0056215 February 2021 Guha
2021/0090694 March 2021 Colley
107403236 November 2017
2002073331 September 2002

Grumberg, et al., Learning to Order BDD Variables in Verification, Journal of Artificial Intelligence Research 18, 2003, pp. 83-116 (Year: 2003). cited by examiner
Jongejan, et al., Automatic Training of Lemmatization Rules That Handle Morphological Changes in pre- , in-and Suffixes Alike, Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, 2009, pp. 145-153, (Year: 2009). cited by examiner
IBM, “List of IBM Patents or Patent Applications Treated As Related,” for U.S. Appl. No. 16/666,747, filed Oct. 29, 2019, 2 pages. cited by applicant
U.S. Appl. No. 16/666,677 “Alternative Modalities Generation for Digital Content Based on Presentation Context,” filed Oct. 29, 2019. cited by applicant
Truong Cong Thang et al., “Effective Adaptation of Multimedia Documents with Modality Conversion”, Signal Processing: Image Communication, vol. 20, Issue 5, Jun. 2005. cited by applicant
Groger, Christoph, et al., “The Manufacturing Knowledge Repository,” Consolidating Knowledge to Enable Holistic Process Knowledge Management in Manufacturing, Proceedings of the 16th International Conference on Enterprise Information Systems (ICEIS), 2014, 13 pages. cited by applicant
Gong, Dihong, “Towards Building Large-Scale Multimodal Knowledge Bases,” pp. 2-35. cited by applicant
Li, Huadong, et al., “Multimodal Question Answering over Structured Data with Ambiguous Entities,” Proceedings of the 26th International Conference on World Wide Web Companion, pp. 79-88, International World Wide Web Conferences Steering Committee, 2017. cited by applicant
Qi, Fan, et al., “A Unified Framework for Multimodal Domain Adaptation,” 2018 ACM Multimedia Conference on Multimedia Conference, pp. 429-437, ACM, 2018. cited by applicant
Vassev, Emil, et al.“Requirements and initial model for knowlang: a language for knowledge representation in autonomic service-component ensembles.” Proceedings of The Fourth International C Conference on Computer Science and Software Engineering. 2011. cited by applicant
Dolog, Peter, et al. “Integrating Adaptive Hypermedia Techniques and Open RDF-based Environments.” WWW (Alternate Paper Tracks). 2003. cited by applicant
Tran, Thanh, Philipp Cimiano, and Anupriya Ankolekar. “Rules for an Ontology-based Approach to Adaptation.” 2006 First International Workshop on Semantic Media Adaptation and Personalization (SMAP'06). IEEE, 2006. cited by applicant

edspgr.12260345