So lately I have been working with smart cards that hold some information and what I am trying to achieve here is to fetch this data from these smart card using a smart card
Typical USB smartcard readers implement the USB CCID device class specification. Consequently, you need to implement that protocol in your application in order to communicate with the reader (and the card). See Communicate with smartcard reader through Android USB host for a (partially working) starting point on how to implement that.