Readers

Methods and coroutines related to readers abstractions. More...

Namespaces
namespace	mtcore::io::readers
	Additional algorithms that can be done on a reader without having to extend the reader type We're encouraging this method of extending readers to avoid readers from becoming incredibly large, and to create a standardized extension mechanism to allow 3rd party extensions of readers.

Classes
struct	mtcore::io::Reader< Impl >
	A reader that reads data from some sort of stream or buffer Note: the data elements read should be trivially constructible, trivially destructible, and trivially copyable. More...

Functions
template<typename T>
Result< typename Reader< T >::ResSlice, typename Reader< T >::ErrType >	mtcore::io::readers::read_until_or_eof (Reader< T > &self, typename Reader< T >::BuffSlice buff, const std::remove_const_t< typename Reader< T >::ReadElem > &delim)
	Reads data from a reader into a Slice up until there is either a delimiter or end of file If the buffer is not big enough, the error SIZE_EXCEEDED will be returned.
template<typename T>
Result< typename Reader< T >::BuffSlice, typename Reader< T >::ErrType >	mtcore::io::readers::read_until_or_eof (Reader< T > &self, Allocator &alloc, const std::remove_const_t< typename Reader< T >::ReadElem > &delim, size_t maxSize=std::numeric_limits< size_t >::max())
	Reads data from a reader into an allocated Slice up until there is either a delimiter or end of file If the buffer is not big enough, the error SIZE_EXCEEDED will be returned Will return a Slice that the caller MUST clean up with alloc.destroy_many() Will try to shrink internal buffer before returning.
template<typename T>
Result< typename Reader< T >::ResSlice, typename Reader< T >::ErrType >	mtcore::io::readers::read_until_no_eof (Reader< T > &self, typename Reader< T >::BuffSlice buff, const std::remove_const_t< typename Reader< T >::ReadElem > &delim)
	Reads data from a reader into a Slice up until there is either a delimiter If the buffer is not big enough, the error SIZE_EXCEEDED will be returned If the end of file is reached, the error END_OF_FILE will be returned.
template<typename T>
Result< typename Reader< T >::BuffSlice, typename Reader< T >::ErrType >	mtcore::io::readers::read_until_no_eof (Reader< T > &self, Allocator &alloc, const std::remove_const_t< typename Reader< T >::ReadElem > &delim, size_t maxSize=std::numeric_limits< size_t >::max())
	Reads data from a reader into an allocated Slice up until there is either a delimiter If the buffer is not big enough, the error SIZE_EXCEEDED will be returned If the end of file is reached, the error END_OF_FILE will be returned Will return a Slice that the caller MUST clean up with alloc.destroy_many() Will try to shrink internal buffer before returning.
template<typename T>
Reader< impl::SliceReaderImpl< T > >	mtcore::io::slice_reader (Slice< T > buff)
	Creates a reader to read the contents of a Slice.

Detailed Description

Methods and coroutines related to readers abstractions.

Function Documentation

read_until_no_eof() [1/2]

template<typename T>

Result< typename Reader< T >::BuffSlice, typename Reader< T >::ErrType > mtcore::io::readers::read_until_no_eof	(	Reader< T > &	self,
		Allocator &	alloc,
		const std::remove_const_t< typename Reader< T >::ReadElem > &	delim,
		size_t	maxSize = std::numeric_limits<size_t>::max() )

Reads data from a reader into an allocated Slice up until there is either a delimiter If the buffer is not big enough, the error SIZE_EXCEEDED will be returned If the end of file is reached, the error END_OF_FILE will be returned Will return a Slice that the caller MUST clean up with alloc.destroy_many() Will try to shrink internal buffer before returning.

Template Parameters

T	Type of data being read

Parameters

self	Reader to operate on
alloc	Allocator to use to create buffer
delim	Delimiter to read until
maxSize	Maximum size of data to be read (anything >100MB will get smaller allocations)

Returns

Result of data read, or error

Definition at line 500 of file io/reader.hpp.

                                                                       {
      using ErrType   = typename Reader<T>::ErrType;
      using BuffSlice = typename Reader<T>::BuffSlice;
      using ReadElem  = typename Reader<T>::ReadElem;
      ensure(maxSize > 0, "BAD MAX SIZE");
      // If we have no limit or a really high limit (> 100MB), we're going to allocate in chunks
      if (maxSize == std::numeric_limits<size_t>::max() || maxSize >= 100000000) {
        // A segmented list will be use to track the data
        SegmentedList<std::remove_const_t<ReadElem>> fullBuff;
        if (auto res = fullBuff.init(alloc, 4000); res.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
        // We will do a full copy before a successful return
        // So we can safely set it up to always clean up the full buff
        mtdefer { fullBuff.deinit(alloc); };
 
        size_t count = 0;
 
        while (true) {
          auto readRes = self.read_one();
          if (readRes.is_error()) {
            if (readRes.error().code == ErrType::END_OF_FILE) {
              return error(ErrType::END_OF_FILE);
            }
            return error(ErrType::ALLOCATION_FAILED);
          }
 
          auto cur = readRes.value();
          if (cur == delim) {
            break;
          }
          if (auto pushRes = fullBuff.push(alloc, cur); pushRes.is_error()) {
            return error(ErrType::ALLOCATION_FAILED);
          }
          ++count;
        }
 
        if (count == 0) {
          return BuffSlice{nullptr, 0};
        }
 
        // We have everything chunked in memory, but our contract is to return a contiguous set of bytes
        // To do this, we need to do one final copy into contiguous memory
        auto finalRes = alloc.create_many<std::remove_const_t<ReadElem>>(count);
        if (finalRes.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
 
        auto finalSlice = finalRes.value();
        // We'll use a Slice writer to make things a little simpler
        auto finalWriter = slice_writer(finalSlice);
        // Since we stored the chunks, we can just iterate each chunk and write the entire chunk at once
        auto iter = fullBuff.iter();
        std::remove_const_t<ReadElem> curElem;
        while (iter.next().copy_if_present(curElem)) {
          if (auto r = finalWriter.write(curElem); r.is_error()) {
            alloc.destroy_many(finalSlice);
            return error(ErrType::ALLOCATION_FAILED);
          }
        }
 
        // Return our Result
        return finalSlice;
      }
      else {
        // If we have a smal enough max size, just read it all into one chunk of memory
        auto outRes = alloc.create_many<std::remove_const_t<ReadElem>>(maxSize);
        if (outRes.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
        auto out = outRes.value();
        auto res = read_until_no_eof(self, out, delim);
        if (res.is_error()) {
          alloc.destroy_many(out);
          return res.error();
        }
        auto finalRes = res.value();
 
        // Check to see if we can save memory if we shrink things down
        // This is more CPU time, but less memory pressure
        if (finalRes.size() + alignof(ReadElem) < out.size()) {
          if (finalRes.size() == 0) {
            alloc.destroy_many(out);
            return BuffSlice{nullptr, 0};
          }
          // Try to create a sub optimal buffer
          auto smallerRes = alloc.create_many<std::remove_const_t<ReadElem>>(finalRes.size());
          if (smallerRes.is_error()) {
            // we have it already in memory, it's just suboptimal. Return it anyway
            return out.sub(0, finalRes.size());
          }
          auto smaller = smallerRes.value();
          auto w       = slice_writer(smaller);
          ensure(w.write_all(finalRes).is_success(), "SOMETHING IS WRONG WITH SLICE WRITER!");
          alloc.destroy_many(out);
          return success(smaller);
        }
        return success(out.sub(0, finalRes.size()));
      }
    }

Here is the call graph for this function:

read_until_no_eof() [2/2]

template<typename T>

Result< typename Reader< T >::ResSlice, typename Reader< T >::ErrType > mtcore::io::readers::read_until_no_eof	(	Reader< T > &	self,
		typename Reader< T >::BuffSlice	buff,
		const std::remove_const_t< typename Reader< T >::ReadElem > &	delim )

Reads data from a reader into a Slice up until there is either a delimiter If the buffer is not big enough, the error SIZE_EXCEEDED will be returned If the end of file is reached, the error END_OF_FILE will be returned.

Template Parameters

T	Type of data being read

Parameters

self	Reader to operate on
buff	Buff to write read data to
delim	Delimiter to read until

Returns

Result of data read, or error

Definition at line 457 of file io/reader.hpp.

                                                                                  {
      using ErrType = typename Reader<T>::ErrType;
      for (size_t i = 0; i < buff.size(); ++i) {
        auto chRes = self.read_one();
        if (chRes.is_error()) {
          if (chRes.error().code == ErrType::END_OF_FILE) {
            return error(ErrType::END_OF_FILE);
          }
          return chRes.error();
        }
        auto ch = chRes.value();
        if (ch == delim) {
          return buff.sub(0, i).to_const();
        }
        buff[i] = ch;
      }
      auto nxt = self.read_one();
      if (nxt.is_error()) {
        return nxt.error();
      }
      if (nxt.value() != delim) {
        return error(ErrType::SIZE_EXCEEDED);
      }
      return buff.to_const();
    }

Here is the call graph for this function:

Here is the caller graph for this function:

read_until_or_eof() [1/2]

template<typename T>

Result< typename Reader< T >::BuffSlice, typename Reader< T >::ErrType > mtcore::io::readers::read_until_or_eof	(	Reader< T > &	self,
		Allocator &	alloc,
		const std::remove_const_t< typename Reader< T >::ReadElem > &	delim,
		size_t	maxSize = std::numeric_limits<size_t>::max() )

Reads data from a reader into an allocated Slice up until there is either a delimiter or end of file If the buffer is not big enough, the error SIZE_EXCEEDED will be returned Will return a Slice that the caller MUST clean up with alloc.destroy_many() Will try to shrink internal buffer before returning.

Template Parameters

T	Type of data being read

Parameters

self	Reader to operate on
alloc	Allocator to use to create buffer
delim	Delimiter to read until
maxSize	Maximum size of data to be read (anything >100MB will get smaller allocations)

Returns

Result of data read, or error

Definition at line 342 of file io/reader.hpp.

                                                                       {
      using ErrType   = typename Reader<T>::ErrType;
      using BuffSlice = typename Reader<T>::BuffSlice;
      using ReadElem  = typename Reader<T>::ReadElem;
      ensure(maxSize > 0, "BAD MAX SIZE");
      // If we have no limit or a really high limit (> 100MB), we're going to allocate in chunks
      if (maxSize == std::numeric_limits<size_t>::max() || maxSize >= 100000000) {
        // A segmented list will be use to track the data
        SegmentedList<std::remove_const_t<ReadElem>> fullBuff;
        if (auto res = fullBuff.init(alloc, 4000); res.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
        // We will do a full copy before a successful return
        // So we can safely set it up to always clean up the full buff
        mtdefer { fullBuff.deinit(alloc); };
 
        size_t count = 0;
 
        while (true) {
          auto readRes = self.read_one();
          if (readRes.is_error()) {
            if (readRes.error().code == ErrType::END_OF_FILE) {
              break;
            }
            return error(ErrType::ALLOCATION_FAILED);
          }
 
          auto cur = readRes.value();
          if (cur == delim) {
            break;
          }
          if (auto pushRes = fullBuff.push(alloc, cur); pushRes.is_error()) {
            return error(ErrType::ALLOCATION_FAILED);
          }
          ++count;
        }
 
        if (count == 0) {
          return BuffSlice{nullptr, 0};
        }
 
        // We have everything chunked in memory, but our contract is to return a contiguous set of bytes
        // To do this, we need to do one final copy into contiguous memory
        auto finalRes = alloc.create_many<std::remove_const_t<ReadElem>>(count);
        if (finalRes.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
 
        auto finalSlice = finalRes.value();
        // We'll use a Slice writer to make things a little simpler
        auto finalWriter = slice_writer(finalSlice);
        // Since we stored the chunks, we can just iterate each chunk and write the entire chunk at once
        auto iter = fullBuff.iter();
        std::remove_const_t<ReadElem> curElem;
        while (iter.next().copy_if_present(curElem)) {
          if (auto r = finalWriter.write(curElem); r.is_error()) {
            alloc.destroy_many(finalSlice);
            return error(ErrType::ALLOCATION_FAILED);
          }
        }
 
        // Return our Result
        return finalSlice;
      }
      else {
        // If we have a smal enough max size, just read it all into one chunk of memory
        auto outRes = alloc.create_many<std::remove_const_t<ReadElem>>(maxSize);
        if (outRes.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
        auto out = outRes.value();
        auto res = read_until_or_eof(self, out, delim);
        if (res.is_error()) {
          alloc.destroy_many(out);
          return res.error();
        }
        auto finalRes = res.value();
 
        // Check to see if we can save memory if we shrink things down
        // This is more CPU time, but less memory pressure
        if (finalRes.size() + alignof(ReadElem) < out.size()) {
          if (finalRes.size() == 0) {
            alloc.destroy_many(out);
            return BuffSlice{nullptr, 0};
          }
          // Try to create a sub optimal buffer
          auto smallerRes = alloc.create_many<std::remove_const_t<ReadElem>>(finalRes.size());
          if (smallerRes.is_error()) {
            // we have it already in memory, it's just suboptimal. Return it anyway
            return out.sub(0, finalRes.size());
          }
          auto smaller = smallerRes.value();
          auto w       = slice_writer(smaller);
          ensure(w.write_all(finalRes).is_success(), "SOMETHING IS WRONG WITH SLICE WRITER!");
          alloc.destroy_many(out);
          return success(smaller);
        }
        return success(out.sub(0, finalRes.size()));
      }
    }

Here is the call graph for this function:

read_until_or_eof() [2/2]

template<typename T>

Result< typename Reader< T >::ResSlice, typename Reader< T >::ErrType > mtcore::io::readers::read_until_or_eof	(	Reader< T > &	self,
		typename Reader< T >::BuffSlice	buff,
		const std::remove_const_t< typename Reader< T >::ReadElem > &	delim )

Reads data from a reader into a Slice up until there is either a delimiter or end of file If the buffer is not big enough, the error SIZE_EXCEEDED will be returned.

Template Parameters

T	Type of data being read

Parameters

self	Reader to operate on
buff	Buff to write read data to
delim	Delimiter to read until

Returns

Result of data read, or error

Definition at line 297 of file io/reader.hpp.

                                                                                  {
      using ErrType = typename Reader<T>::ErrType;
      for (size_t i = 0; i < buff.size(); ++i) {
        auto chRes = self.read_one();
        if (chRes.is_error()) {
          if (chRes.error().code == ErrType::END_OF_FILE) {
            return buff.sub(0, i).to_const();
          }
          return chRes.error();
        }
        auto ch = chRes.value();
        if (ch == delim) {
          return buff.sub(0, i).to_const();
        }
        buff[i] = ch;
      }
      auto nxt = self.read_one();
      if (nxt.is_error()) {
        if (nxt.error().code == ErrType::END_OF_FILE) {
          return buff.to_const();
        }
        return nxt.error();
      }
      if (nxt.value() != delim) {
        return error(ErrType::SIZE_EXCEEDED);
      }
      return buff.to_const();
    }

Here is the call graph for this function:

Here is the caller graph for this function:

slice_reader()

template<typename T>

Reader< impl::SliceReaderImpl< T > > mtcore::io::slice_reader

(

Slice< T >

buff

)

Creates a reader to read the contents of a Slice.

Template Parameters

T	Data type to read (should be trivially copyable, constructible, and destructible)

Parameters

buff	Buffer to read from

Returns

A reader that reads from the Slice buffer

Definition at line 646 of file io/reader.hpp.

                                                             {
    return Reader<impl::SliceReaderImpl<T>>{.underlying = impl::SliceReaderImpl<T>{.buff = buff}};
  }